Bitmap index compression

ABSTRACT

A method and apparatus for compressing data is provided. The invention compresses an input bit stream into a compressed output bit stream. The input bit streams are byte aligned and classified. Bytes with all bits set to value zero are classified as gap bytes. Bytes with only one bit set to value one are classified as offset bytes. All other bytes are classified as map bytes. 
     Groups of adjacent bytes are organized into two types of groups. The first type is a gap bit group. A gap map group contains gap bytes and one offset byte. The second type is the gap map group. It contains gap bytes and map bytes. The number of gap bytes in a group is called a gap size. 
     The groups are compressed into four types of atoms. Each type of atom has one control byte, zero or more gap size bytes, and zero or map bytes. A control byte describes the atom. The map bytes in an atom are copies of the map bytes in the control group.

RELATED APPLICATIONS

This application is a continuation of application Ser. No. 08/808,560filed Feb. 28, 1997, now U.S. Pat. No. 5,907,297.

The present application is related to: U.S. patent application Ser. No.08/807,334, entitled “CREATING BITMAPS FROM MULTI-LEVEL IDENTIFIERS”,filed by Cetin Ozbutun, Michael Depledge, Hakan Jakobsson, Mark Kremer,Jeffrey I. Cohen, Quoc Tai Tran, and Alexander C. Ho on Feb. 28, 1997the contents of which are incorporated herein by reference.

U.S. patent application Ser. No. 08/808,584, entitled “BITMAPSEGMENTATION”, filed by Cetin Ozbutun, Jeffrey I. Cohen, HakanJakobsson, Mark Kremer, Michael Depledge, Quoc Tai Tran, Alexander C.Ho, and Julian Hyde, on the Feb. 28, 1997 the contents of which areincorporated herein by reference.

U.S. patent application Ser. No. 08/752,128, entitled “METHOD ANDAPPARATUS FOR PROCESSING COUNT STATEMENTS IN A DATABASE SYSTEM”, filedby Cetin Ozbutun, Michael Depledge, Hakan Jakobsson, and Jeffrey I.Cohen, on Nov. 20, 1996, the contents of which are incorporated hereinby reference.

U.S. patent application Ser. No. 08/808,097, entitled “GROUP BY ANDDISTINCT SORT ELIMINATION USING COST-BASED OPTIMIZATION”, filed byJeffrey Ira Cohen, Cetin Ozbutun, Michael Depledge, and Hakan Jakobsson,on Feb. 28, 1997 the contents of which are incorporated herein byreference.

U.S. patent application Ser. No. 08/808,096, entitled “METHOD ANDAPPARATUS FOR USING INCOMPATIBLE TYPES OF INDEXES TO PROCESS A SINGLEQUERY”, filed by Jeffrey Ira Cohen, Cetin Ozbutun, Hakan Jakobsson, andMichael Depledge, on Feb. 28, 1997 the contents of which areincorporated herein by reference.

U.S. patent application Ser. No. 08/808,094, entitled “INDEX SELECTIONFOR AN INDEX ACCESS PATH”, filed by Hakan Jakobsson, Michael Depledge,Cetin Ozbutun, and Jeffrey I. Cohen, on Feb. 28, 1997 the contents ofwhich are incorporated herein by reference.

U.S. patent application Ser. No. 08/807,429, entitled “QUERY PROCESSINGUSING COMPRESSED BITMAPS”, filed by Cetin Ozbutun, Jeffry I. Cohen,Michael Depledge, Julian Hyde, Hakan Jakobsson, Mark Kremer, and QuocTai Tran, on Feb. 28, 1997 the contents of which are incorporated hereinby reference.

U.S. patent application Ser. No. 08/807,451, entitled “BITMAPPEDINDEXING WITH HIGH GRANULARITY LOCKING”, filed by Michael Depledge,Jeffrey I. Cohen, Hakan Jakobsson, Mark Kremer, Cetin Ozbutun, Quoc TaiTran, and Alexander C. Ho, on Feb. 28, 1997 the contents of which areincorporated herein by reference.

U.S. patent application Ser. No. 08/808,560, entitled “UPDATINGBITMAPPED INDEXES”, filed by Michael Depledge, Hakan Jakobsson, CetinOzbutun, Jeffrey I. Cohen, and Quoc Tai Tran, on Feb. 28, 1997 thecontents of which are incorporated herein by reference.

U.S. patent application Ser. No. 08/808,586, entitled “COMBINING BITMAPSWITHIN A MEMORY LIMIT”, filed by Cetin Ozbutun, Jeffry I. Cohen, MichaelDepledge, Julian Hyde, Hakan Jakobsson, Mark Kremer, and Quoc Tai Tran,on Feb. 28, 1997 the contents of which are incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to a method of compressing data incomputer systems, and more particularly, to the compression of bitmapswithin bitmap indexes used to access data stored in databases.

BACKGROUND OF THE INVENTION

A bitmap index is an index that includes a set of bitmaps that can beused to efficiently process queries on a body of data associated withthe bitmap index. In the context of bitmap indexes, a bitmap is a seriesof bits that indicate which of the records stored in the body of datasatisfy a particular criteria. Each record in the body of data has acorresponding bit in the bitmap. Each bit in the bitmap serves as a flagto indicate whether the record that corresponds to the bit satisfies thecriteria associated with the bitmap.

Typically, the criteria associated with a bitmap is whether thecorresponding records contain a particular key value. In the bitmap fora given key value, all records that contain the key value have theircorresponding bits set to 1 while all other bits are set to 0. Acollection of bitmaps for the key values that occur in the data recordscan be used to index the data records. In order to retrieve the datarecords with a given key value, the bitmap for that key value isretrieved from the index, and for each bit set to 1 in the bitmap, thecorresponding data record is retrieved. The records that correspond tobits are located based on a mapping function between bit positions anddata records.

Since bitmaps are in the form of binary numbers, they can be combined inlogical operations, such as AND operations, very efficiently in adigital computer. However, bitmaps waste space when a large portion ofeach bitmap is used to store nothing but logical zeros. For example,assume that a table contains a million rows, where a particular columnof the table has 500,000 distinct values. A bitmap index on that columnwould have 500,000 index entries storing bitmaps which, on average, havetwo bits set to “1” and 999,998 bits set to “0”.

To further enhance the efficiency of bitmaps, especially those withlarge sequences of logical zeros, compression is used. There are manycompression techniques. However, none of the known compressiontechniques is designed specifically for the distribution of bits foundin the bitmaps of bitmap indexes found in large databases.

One example of a compression method is described in U.S. Pat. No.5,363,098 entitled “Byte Aligned Data Compression,” issued to GennadyAntoshenkov on Nov. 8, 1994. In general terms, the '098 method dividesbytes into two classes. The first class is Gap bytes (GBYTES), which arebytes with all the bits set to the same value, either logical one orlogical zero. The second class is Map bytes (MBYTE), which are byteswhere all the bits are not set to the same value. Finally, the number ofbytes in a sequence of consecutive GBYTES is called the gap size.

The '098 method represents groups of consecutive MBYTES or groups ofconsecutive GBYTES (followed optionally by MBYTES) as encoded atoms ofbytes. The first byte is referred to as the control byte. The controlbyte (CBYTE) describes the bytes that are in the atom. In atoms thatrepresent MBYTES, the MBYTES themselves follow the CBYTE at some point.In atoms that represent GBYTES, bytes may or may not follow the CBYTE.

The CBYTE is divided into three fields of adjacent bits, which are theTFIELD, FFIELD, and the DFIELD. The use of the fields depends on whetherMBYTES or GBYTES are being encoded in the atom. The TFIELD is a threebit field denoting the type of atom, including special case atoms. TheFFIELD is used to denote the value of the bits in GBYTES, or in otherwords, whether all the bits are set to one or zero. The DFIELD is eithera three or four bit field denoting the number of MBYTES in the atom.

In atoms encoding GBYTES, the gap size is used to indicate how manyGBYTES are represented by the atom. Gap size is an integer number. Forthe smaller gap sizes, the gap size is represented by the TFIELD. Whenthis field ranges in value from 0 to 3, it represents a gap size of 1 to4 bytes respectively. For gap sizes greater than 1 to 4, gap size isstored in a series of bytes which immediately follow the CBYTE. Largerseries are needed for larger gap sizes. The first byte in a series usesa field of adjacent bits to represent the number of bytes in the series.The rest of the bits in the first byte and in the following bytes can beused to specify the gap size.

The '098 method is less than optimal for representing the GBYTES ofbitmap indexes in databases. In typical bitmap indexes, almost all theGBYTES have bits of zero value. Thus, using the FFIELD to denote thevalue can be wasteful because most GBYTES have bits of zero valueanyway. Further, in bitmap indexes found in databases, the gap size isskewed towards the smaller numbers. The '098 method waste bits thatcould be used to represent smaller numbers for gap sizes. The FFIELD bitis wasted by using the bit to denote the value of the bits in GBYTESrather than using it as an additional bit to represent a number. Incases where a series of bytes following the CBYTE is used to representthe gap size, some of the bits of the first byte in the series arewasted by using them to represent the number in the series rather thangap sizes.

Based on the foregoing, it is clearly desirable to provide a mechanismthat is better adapted from compressing the bit distribution found inthe bitmaps within the bitmap indexes of databases. It is furtherdesirable to provide a mechanism that is better adapted for compressingdata composed of small-sized gaps of zero value GBYTES.

SUMMARY OF THE INVENTION

The invention compresses an input bit stream into a compressed outputbit stream. The input bit streams are byte aligned and classified. Byteswith all bits set to value zero are classified as gap bytes. Bytes withonly one bit set to value one are classified as offset bytes. All otherbytes are classified as map bytes.

Groups of adjacent bytes are organized into two types of groups. Thefirst type is a gap bit group. A gap bit group contains gap bytes andone offset byte. The second type is the gap map group. It contains gapbytes and map bytes. The number of gap bytes in a group is called a gapsize.

The groups are compressed into four types of atoms. Each type of atomhas one control byte, zero or more gap size bytes, and zero or mapbytes. A control byte describes the atom. The map bytes in an atom arecopies of the map bytes in the group.

A control byte is composed of two fields. The DFIELD is composed of athree bit sequence and the TFIELD is composed of a five bit sequence.The DFIELD and TFIELD are composed of the same sequence of bits in alltypes of atoms, and each field contains a value. The range in which thevalue in the TFIELD falls indicates the type of atom. The TFIELD alsorepresents the gap size of the atom. The DFIELD is used to eitherindicate which bit in an offset byte is set to value one, or to indicatethe number of map bytes in the atom.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings and in whichlike reference numerals refer to similar elements and in which:

FIG. 1 is a block diagram of a computer system that may be used toimplement an embodiment of the invention;

FIG. 2 shows 4 portions of input streams used to demonstrate thecompression of an embodiment of the invention;

FIG. 3 shows the basic structural element of compressed output referredto as atoms;

FIG. 4 shows two input streams representing gap bit groups compressedinto either a short gap map atom or a long gap map atom;

FIG. 5 shows two input stream representing gap map groups compressedinto either a short gap map group atom or a long gap map group atom;

FIG. 6 shows an example of compressing an input bit stream into a outputbit stream;

FIG. 7 shows a comparison of an alternative method for setting gap sizebytes;

FIG. 8 outlines the steps to compress an input stream into an outputstream; and

FIG. 9 shows a summary of the structure of atoms.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and apparatus for compressing data is described. In thefollowing description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the present invention may be practicedwithout these specific details. In other instances, well-knownstructures and devices are shown in block diagram form in order to avoidunnecessarily obscuring the present invention.

Hardware Overview

Referring to FIG. 1, it is a block diagram of a computer system 100 uponwhich an embodiment of the present invention can be implemented.Computer system 100 includes a bus 101 or other communication mechanismfor communicating information, and a processor 102 coupled with bus 101for processing information. Computer system 100 further comprises arandom access memory (RAM) or other dynamic storage device 104 (referredto as main memory), coupled to bus 101 for storing information andinstructions to be executed by processor 102. Main memory 104 also maybe used for storing temporary variables or other intermediateinformation during execution of instructions by processor 102. Computersystem 100 also comprises a read only memory (ROM) and/or other staticstorage device 106 coupled to bus 101 for storing static information andinstructions for processor 102. Data storage device 107 is coupled tobus 101 for storing information and instructions.

A data storage device 107 such as a magnetic disk or optical disk andits corresponding disk drive can be coupled to computer system 100.Computer system 100 can also be coupled via bus 101 to a display device121, such as a cathode ray tube (CRT), for displaying information to acomputer user. Computer system 100 further includes a keyboard 122 and acursor control 123, such as a mouse.

The present invention is related to the use of computer system 100 tocompressing data According to one embodiment, compressing data isperformed by computer system 100 in response to processor 102 executingsequences of instructions contained in memory 104. Such instructions maybe read into memory 104 from another computer-readable medium, such asdata storage device 107. Execution of the sequences of instructionscontained in memory 104 causes processor 102 to perform the processsteps that will be described hereafter. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions to implement the present invention. Thus, thepresent invention is not limited to any specific combination of hardwarecircuitry and software.

Exemplary Bit Streams

FIG. 2 illustrates portions of input bit streams 210, 220, 230, and 240.These bit streams are used to describe the compression of an embodimentof the invention. The bit streams are from a bitmap within a bitmapindex of a database.

The plurality of bits 211 are shown within their respective bit streamsfrom left to right order. The first bit in bit stream 210 is bit 1 andthe bit is set to value zero. All the bit streams of FIG. 2 begin with abit 1 for purposes of illustration only. The first bit set to value onein bit stream 210 is bit 186. Bit zero and bit 186, like all the otherbits in bit stream 210, each represent a row in a database table. Thefirst bit set to the one value in bit stream 220 is bit 735,690 thefirst bit set to the one value in bit stream 230 is bit 42, and thefirst bit set to value one in bit stream 240 is bit 130.

It should be apparent that the input bit streams 210, 220, 230, and 240can represent bit streams from sources other than bitmaps within thebitmap indexes of databases. For instance, bit streams 210, 220, 230,and 240 could represent a portion of a bitmap of pixels for an image, orrepresent data flow received from a media server.

Division of Stream Into Bytes

FIG. 2 shows a portion of bit stream 210. For compatibility with thehardware illustrated in FIG. 1, data in the form of bits from input bitstreams like input bit stream 210 are aligned into bit sequences withidentical numbers of bits, then stored in computer system 100. A bitsequence is herein after called a byte. Bytes 212, 214, 215, and 216 ininput stream 210 are examples of bytes. In FIG. 4, the illustratedportion of input bit stream 210 is shown as bit stream 410, which is abit stream presented in aligned form. Accordingly, bit stream 410contains bytes 212, 214, 215, and 216.

The bits from input bit streams 220, 230, and 240 are likewise alignedand stored in computer system 100. FIG. 4 shows the illustrated portionof input bit stream 220 in aligned form as bit stream 420. FIG. 5 showsthe illustrated portions of input bit stream 230 and input bit stream240 in aligned form respectively as bit stream 530 and bit stream 540.

The bytes in all the figures are eight bit bytes. It should beunderstood that an embodiment of the invention can just as easily beadapted for computer systems having other byte sizes.

Classification of Bytes

In order to achieve the compression, the bytes in bit streams 410, 420,530, and 540 are first classified. Bytes with all bits set to value zeroare classified as a gap byte (GBYTE). Bytes with just one bit set tovalue one are classified as an offset byte (OBYTE). All other bytes areclassified as a map byte (MBYTE).

FIG. 4 shows an example of a GBYTE as byte 212, and an example of anOBYTE is byte 216. FIG. 5 shows an example of an MBYTE as any of theplurality of MBYTEs 538.

Grouping Bytes

Adjacent bytes are organized into groups. There are two types of groups.The type into which a group is placed depends on the types of bytes thegroup contains. The first type of group is a gap bit group. The gap bitgroup contains zero or more contiguous GBYTEs followed by one OBYTE. Aset of zero or more contiguous GBYTES is called a gap. The number ofGBYTEs in a gap is referred to as gap size. The gap size alsocorresponds to the number of GBYTES in a gap bit group or a gap mapgroup.

FIG. 4 shows bit streams that would be grouped as a gap bit group. Bitstream 410 contains 23 GBYTEs followed by an OBYTE byte 216. Bit stream420 contains 91,161 GBYTEs followed by an OBYTE 424. The gap size of bitstream 410 and bit stream 420 is 23 and 91,161 respectively.

The second type is a gap map group. This type contains zero or moreGBYTEs and zero or more MBYTEs. FIG. 5 shows bit streams that would beplaced in a gap map group. Bit stream 530 contains 5 GBYTEs followed byplurality of 8 MBYTEs 538. Bit stream 540 contains 16 GBYTEs followed byplurality of 8 MBYTEs 545.

Generating Compressed Output Stream

The groups represented by bit streams 410, 420, 530, and 540 arecompressed into output groups hereinafter referred to as atoms. Thereare four types of atoms, which are the short gap bit atom, the long gapbit atom, the short gap bit atom, and the long gap map atom. The type towhich an atom belongs depends on the type of group being compressed andthe gap size of the group. The basic structure of the atom is shown inFIG. 3. Each atom for each group contains one control byte (CBYTE) 310,zero or more gap size bytes 320, and zero or more MBYTEs 330. In FIG. 9,table 900 summarizes details about the atoms which shall be describedbelow.

The CBYTE 310 is composed of two fields used to describe the atom. Thetwo fields are the DFIELD 312 and the TFIELD 314. The DFIELD 312 is a 3bit sequence used to either describe the OBYTE in the group or thenumber of MBYTEs in the group. The TFIELD 314 is a 5 bit sequence usedto both indicate the type of the atom, and, in conjunction with the gapsize bytes 320, to represent the the gap size of the group beingcompressed. The precise use of the DFIELD 312, TFIELD 314, and gap sizebytes 320 shall be explained in further detail.

One advantage of an embodiment of the invention is that a DFIELD and aTFIELD are the same size in every atom type. This uniformity of sizeenables the computer instructions that cause the computer system 100 toextract the DFIELD and TFIELD from the CBYTE to be implemented in amanner more efficient for computer system 100. This efficiency isespecially important because the operation of extracting a DFIELD andTFIELD is highly repeated.

Short Gap Bit Atom

The short gap bit atom is generated to compress a gap bit group when thegap size is 23 or less. The atom is composed of just one byte which is aCBYTE. Representing up to 23 bytes with just one CBYTE is one aspect ofan embodiment of the invention that enables it to achieve compression.This aspect is of particular advantage for compressing the bitmapswithin the bitmap indexes of databases. The gap sizes of the gaps inthese bitmaps are skewed toward the lower numbers like the gap sizesthat can be represented by a short gap bit atom.

The bits in the TFIELD are set to values between 0 and 23. When theTFIELD is set to this range, not only does the TFIELD indicate that theatom is a short gap bit atom, but the TFIELD also represents the gapsize of the group being compressed. The values 0 to 23, when containedin the TFIELD, represent a gap size of 0 to 23, respectively.

The DFIELD is used to describe the offset bit in the group. The value towhich the DFIELD is set indicates which bit is set to value one in theOBYTE.

Using the DFIELD to describe the OBYTE is another aspect of anembodiment of the invention that enables it to achieve compression. Inaddition to using the CBYTE to encode the GBYTEs, an embodiment of theinvention uses the same CBYTE to also encode the OBYTE. This additionaluse of the control byte is of particular advantage for databases.Distributions of gaps followed by an OBYTE are a commonly founddistribution of bits in bitmaps within the bitmap indexes of databases.

In FIG. 4, examples of a short gap bit atoms are shown. Short gap bitatom 404 represents bit stream 402, which is composed one byte, OBYTE403. Short gap bit atom 404 represents the smallest gap size for a shortgap bit atom, gap size 0. Short gap bit atom 404 is composed of onebyte, which is CBYTE 407. The TFIELD 408 contains the number 0 in binaryform. This number represents that there are 0 GBYTEs in bit stream 402.The DFIELD 406 contains the value 6 in binary form. It represents thatthe seventh bit in OBYTE 403 is set to value one.

Short gap bit atom 470 represents bit stream 430, which is composed ofGBYTE 432 and OBYTE 434. Short gap bit atom 470 represents a gap sizeof 1. Short gap bit atom 470 is composed of one byte, which is CBYTE476. The TFIELD 472 contains the number 1 in binary form. This numberrepresents that there is 1 GBYTE in bit stream 430. The DFIELD 474contains the value 6 in binary form. It represents that the seventh bitin OBYTE byte 434 is set to value one.

Short gap bit atom 450 represents the compressed output of bit stream410, and represents the largest gap size that can be represented by onecontrol byte for a short gap bit atom. Short gap bit atom 450 iscomposed of one byte, which is CBYTE 452. The TFIELD 454 contains thenumber 23 in binary form. This number represents that there are 6 GBYTEsin bit stream 410. The DFIELD 454 contains the value 6 in binary form.It represents that the seventh bit in OBYTE byte 216 is set to valueone.

Long Gap Bit Atom

The long gap bit atom is used to compress gap bit groups with gap sizes24 or greater. The atom is composed of one CBYTE and one or more gapsize bytes. The number 24 represents the split. The split is a thresholdnumber that represents the maximum value to which the TFIELD is set foratoms used to compress gap bit groups. The split minus one is thethreshold maximum gap size that can be represented by the one byte inthe short gap bit atom. Beginning at the split, an embodiment of theinvention uses gap size bytes to represent the gap size for gap bitgroups.

The left most bit in a gap size bit is used as a flag to indicatewhether any more gap size bytes follow. Setting this flag to value oneindicates that gap size bytes follow. Setting the flag to value zeroindicates that the gap size byte is the last gap size byte. Theremainder of the bits on the right in the gap size byte are used torepresent a number.

Gap size bytes 320 in FIG. 3 illustrate the use of the gap size bytes.The numbers 0 through 7 in right to left order above gap size byte 326represent a bit position in each gap size byte of gap size bytes 320.Bits 321 in the leading gap size bytes 326 and 327 are set to value oneto indicate a gap size byte follows. Bit 322 in gap size byte 328 is setto value zero to indicate that the gap size byte 328 is the last gapsize byte.

The gap size is represented by a binary number formed by the bits in the0th through 6th positions in gap size bytes 320. The first gap sizebyte, gap size byte 326, contains the 0th through 6th bits in the binarynumber representing the gap size. The next gap size byte contains the7th through 13th bits. The last gap size byte, gap size byte 328,contains the 14th through 20th bits. This pattern of representing thegap size continues for numbers requiring more bits.

Long gap bit atoms have an inherent minimum gap size. The minimum numbercorresponds to the split. To conserve use of bits, an embodiment of theinvention takes advantage of this inherent minimum by storing an offsetto the split in the gap size bytes. To determine the gap size of a gapbit group the split is added to the offset. The advantage of using anoffset is that it is smaller than the gap size and thus requires lessbits to represent. Using less bits within a gap size byte in turn leadsto using less gap size bytes, thus further enhancing compression.

FIG. 4 shows an example of a long gap atom. The long gap bit atom 480represents the compressed output of bit stream 420. Bit stream 420contains 91,961 GBYTEs 422 and OBYTE 424. The long gap bit atom 480contains a CBYTE 486 with a DFIELD 482 and a TFIELD 484. The DFIELD 482,set to value 6, represents that the seventh bit is set to value one inOBYTE 424. The TFIELD 484, set to the value 24, represents that the atomis a long gap atom with gap size bytes 488 following CBYTE 486.

The plurality of left bits 490, set to value one, flag that a gap bytesize follows. Left bit 492, set to value zero, flags that no gap sizebyte follows. The plurality of bits 487 represent the offset 91,937.Adding the offset to the split, the number 24, results in the number91,161.

Long gap bit atom 480 represents, using just 4 bytes, a bit stream 420composed of 91, 962 bytes. Representing with a few bytes a vastly largernumber of bytes is one aspect of an embodiment of the invention thatenables it to achieve compression.

For smaller numbers, the use of the left most bit as a flag conservesthe use of bits in comparison to alternative methods of tracking byteslike gap size bytes. For example, one alternative method of tracking thegap size bytes is to use three bits in the first gap size byte torepresent the number of gap size bytes that follow the first gap sizebyte in a atom.

FIG. 7 is used to compare an embodiment of the invention to thealternative method and shows two sets of gap size bytes in byte alignedform. Gap size bytes 700 represent the alternative method. Gap sizebytes 700 include gap size byte 710 and gap size byte 720. The threebits 712 in gap size byte 710 are used to represent the number of gapsize bytes following gap size byte 710. Because there is one gap sizebyte following gap size byte 710, the three bits 712 are used torepresent the number one. The remaining bits in gap size byte 710, whichare bits 714, are used to represent the number 127. Because bits 714contains five bits, two short of the number of bits needed to representthe number 127, the first two bits in gap size byte 720 are also used.

Gap size byte 754 represents an embodiment of the invention. The gapsize byte 754 contains the number 127 within bits 756. The bit 758 is aflag indicating that gap size byte 754 is the last gap size byte.

To represent the number 127, the alternative method uses 10 bits whilethe invention uses 8, for a difference of 2 bits. The alternative methoduses all 8 bits of the gap size byte 710 plus 2 bits in gap size byte720. The invention uses only the 8 bits in gap size byte 754.Furthermore, the alternate method had to use two bytes in order toobtain the number of bits the method needed to represent the number 127.

For lower numbers like 127, an embodiment of the invention conservesbits, bits which can be used for other purposes such as those shown sofar. Further more, an embodiment of the invention may conserve thenumber bytes used to represent lower numbers.

Short Gap Map Atom

The short gap map atom is used to compress gap map groups with 0 to 5GBYTEs and 1 to 8 MBYTEs. A short gap map atom is composed of a CBYTEand 1 to 8 MBYTEs. The TFIELD ranges in value from 25 to 30. When theTFIELD is set to this range, not only does the TFIELD indicate that theatom is a short gap map atom, but the TFIELD represents the gap size ofthe group. When the TFIELD contains values 25 through 30, the TFIELDrepresents a gap size from 0 to 5 respectively. The DFIELD contains avalue from 0 to 7 which represents a number of MBYTEs ranging from 1through 8 respectively. The MBYTEs in the short gap map atom are copiesof the MBYTEs in the gap map group the atom represents.

FIG. 5 shows an example of a short gap map atom. Short gap map atom 550is a compressed representation of bit stream 530. The CBYTE 552 iscomposed of DFIELD 554 and TFIELD 556. The DFIELD 554 contains the value7 in binary form which indicates that the gap map atom represents 8MBYTEs. The TFIELD contains the value 30 which indicates that the atomrepresents a gap size of 5. The MBYTEs 558 are copies of MBYTEs 538.

CBYTE 552 represents the 5 GBYTEs 532 in bit stream 530. Using one CBYTEto represent all the GBYTEs of short map group is one aspect of thisinvention that enables it to achieve compression.

Long Gap Map Atom

The long gap map atom is used to compress gap map groups that contain 6or more GBYTEs. The atom is composed of a CBYTE, 1 or more gap sizebytes, and 1 to 8 MBYTEs. The DFIELD in the CBYTE indicates the numberof MBYTEs in the long gap map atom in the same manner as the DFIELD in ashort gap atom indicates the number of MBYTEs in the short gap atom. TheTFIELD in the CBYTE contains the value 31, which denotes that the atomis a long gap map atom with gap size bytes. The gap size bytes are usedto indicate the gap size represented by the long gap map atom. These gapsize bytes are used in the same manner as the gap size bytes in the longgap bit atom. The only difference is that the number added to the offsetis 6. This number is the inherent minimum gap size represented by a longgap map atom.

FIG. 5 shows an example of a long gap map atom. The long gap map atom580 is a compressed representation of bit stream 540. The CBYTE 582contains DFIELD 584 and TFIELD 586. The DFIELD is set to the value 7which denotes that long gap map atom 580 represents 8 MBYTEs. The TFIELDcontains value 31 to indicate that gap size bytes are used to indicatethe gap size represented by long gap map atom 580. The MBYTEs 585 inlong gap map atom 580 are copies of MBYTEs 545 in bit stream 540.

The gap size byte 588 contains left bit 589 which is set to value zeroto indicate that no gap size byte follows. The remainder of the bits ingap size byte 588 represent the offset value of 10. Adding the number 6to the offset results in the gap size represented by long gap map atom580, which is 16.

CBYTE 582 and gap size byte 588 represent the 16 GBYTEs 542 in bitstream 540. Using the CBYTE 582 and gap size byte 588 to represent allthe GBYTEs of a gap map group demonstrates one aspect of this inventionthat enables it to achieve compression.

Compression of an Input Stream to an Output Stream

FIG. 8 outlines the steps undertaken to compress an input stream into anoutput stream. In step 810, bits from an input bit stream are stored asaligned bytes and classified as either CBYTEs, MBYTEs, or OBYTEs. Instep 820, adjacent bytes from the input stream are organized into a gapbit group or gap map group. In step 830, a determination is made ofwhether the group is a gap bit group or is otherwise a gap map group.

If the group is a gap bit group, then the next step is step 840. In step840, a determination is made of whether the group has a gap size greaterthan or equal to the threshold represented by the split. If the gap sizeis less than then the split, then the next step is 844. In step 844, ashort gap map atom is generated. If the gap size is greater than orequal to the split, then the next step is 848. In step 848, a long gapbit atom is generated.

If the determination made in step 830 is that the group is a gap mapgroup, then the next step is step 850. In step 850, a determination ismade of whether the group has a gap size is greater than the threshold29 minus the split 24, which is 5. 29 minus the split is the maximum gapsize represented by a short gap map atom. If the gap size is less thanor equal to this threshold, then the next step is 854. In step 854, ashort gap map atom is generated. If the gap size is greater than thisthreshold, then the next step is 858. In step 858, a long gap bit atomis generated.

The bit streams shown in FIG. 6 are used as examples of compressing aninput bit stream into an output bit stream. FIG. 6 shows exemplary inputbit stream 610 and exemplary output bit stream 640. Bit stream 620 is abyte aligned representation of input bit stream 610.

Gap map group 622 is compressed into short gap map atom 642 in outputbit stream 640. The gap map group 622 is composed of two MBYTEs. Thefirst byte in short gap map atom 642 is a CBYTE. The first three bits inthis CBYTE comprise a DFIELD representing the value 2, which is thenumber of MBYTEs represented by short gap map atom 642. The remainingbytes in the CBYTE comprise the TFIELD. This TFIELD represents the value25 which indicates that no GBYTEs follow. The next two bytes in shortgap map atom 642 are MBYTES which are copies of the MBYTES in gap mapgroup 622.

The next group to be compressed is the gap map group 624. Gap map group624 is compressed into long gap map atom 644 in output bit stream 640.Gap map group 624 contains 10 GBYTEs followed by 2 MBYTEs. The firstbyte in the long gap map atom 644 is a CBYTE. The first three bits inthis CBYTE comprise a DFIELD representing the value of 2, which is thenumber of MBYTEs represented by long gap map atom 644. The remainingbits in the CBYTE are the TFIELD. The TFIELD contains the value 31 toindicate that at least one gap size byte follows the CBYTE.

One gap size byte does follow the CBYTE. The left most bit in the gapsize byte is set to value zero to indicate that this byte is the lastgap size byte. The gap size byte contains the offset value 4. Adding 6to the offset results in the number 10, which is the number of gap bytesrepresented by long gap map atom 644. The last two bytes in long gap mapatom 644 are MBYTEs which are copies of the MBYTEs in gap map group 624.

The last group to be compressed is the gap bit group 626. Gap bit group626 is compressed into short gap bit atom 646. Gap bit group 626contains two GBYTEs followed by an OBYTE. Short gap bit atom 646 iscomposed of only one CBYTE. The first three bits are the DFIELD. TheDFIELD contains the number 6 to represent that the 7th bit in the OBYTEis set to the value one. The remainder of the bits are the TFIELD. TheTFIELD is set to the value 2. This value indicates that the short gapbit atom 646 is a short gap bit atom, and that short gap bit atom 646represents 2 GBYTEs.

Adjusting the Split

The split is set to 24 to achieve higher compression for thedistribution of bits found in the bitmaps within the bitmap indexes ofdatabases. In these kind of bitmaps, the distribution is skewed towardsgaps being followed by an OBYTE rather than a MBYTE. In other words, thedistribution is skewed towards gap bit groups rather than gap mapgroups. Short gap bits can represent such gap bit groups with one CBYTEup to gap sizes corresponding to the split minus 1. Increasing the splitpermits larger gap bit groups to be compressed into short gap bit atoms,thus enhancing compression in bitmaps skewed towards gap bit groups.

If the distribution is skewed toward the other direction in favor of gapmap groups, an embodiment of the invention can be adjusted to achievehigher compression for such a distribution. Decreasing the split wouldpermit gap map groups with larger gap sizes to be represented by shortgap map atoms, thus enhancing compression in bitmaps skewed toward gapmap groups.

The present invention offers advantages over prior approaches forcompressing bit steams, especially those from bitmaps within the bitmapindexes of databases. First, an embodiment of the invention is skewedtoward efficiently compressing small gap sizes. Because a gap size up to23 can be represented by one CBYTE, a broader range of small gap sizescan be compressed into one byte. Furthermore, an embodiment of theinvention compresses the more common bit distribution of gaps followedby an offset more efficiently than the less common distribution of gapsfollowed by MBYTES.

The gap size bytes are used efficiently, especially for lower numbers.Only the smaller offset number is stored in the gap size bytes. Usingone bit in the gap size byte as a flag enables an embodiment of theinvention to represent smaller gap sizes more efficiently.

Finally the size of the TFIELD and DFIELD is uniform for all atom types.This uniformity enables implementing an embodiment of the invention intocomputer instructions which are more efficient.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A computer-readable media containing compressedoutput, comprising: compressed sets of one or more bit sequences thatrepresent groups of bit sequences; wherein each bit sequence of saidgroups of bit sequences contains an identical number of bits; whereineach group of bit sequences of said groups of bit sequences includeseither a gap bit sequence, an offset bit sequence, or a map bitsequence; and wherein at least one compressed set of one or more bitsequences of said compressed sets contains a set of one or more gap sizebit sequences, wherein each gap size bit sequence of said set of one ormore gap size bit sequences includes a flag, wherein said flag indicateswhether a subsequent gap size bit sequence follows within said at leastone compressed set.
 2. The computer-readable media of claim 1, whereinwherein each gap bit sequence in said group of bit sequences has allbits set to logical zero, wherein each offset bit sequence in said groupof bit sequences has one bit of said bit sequence set to logical one,wherein each said map bit sequence in said group of bit sequences hastwo or more bits.
 3. The computer-readable media of claim 1, whereineach gap size sequence in said compressed sets of one or more bitsequences stores a portion of a gap size of the respective group of bitsequences of said each gap size sequence.
 4. The computer-readable mediaof claim 1, wherein each compressed set of one or more bit sequences ofsaid compressed sets includes a control bit sequence, wherein saidcontrol bit sequence indicates: a number of gap bit sequences in saideach group, a number of map bit sequences in said each group, and ifsaid each group contains an offset bit sequence, then which bit is setto value one in said offset bit sequence in said each group.
 5. Thecomputer-readable media of claim 1, wherein the compressed sets of oneor more bit sequences include: a first gap bit group that represents afirst group of bit sequences from said groups of bit sequences, whereinsaid first group has a first gap size, wherein said first gap bit groupcontains zero or more offset bit sequences and zero or more gap bitsequences, wherein said first gap size is less than a threshold; and asecond gap bit group that represents a second group of bit sequencesthat has a second gap size, wherein said second gap bit group containszero or more offset bit sequences, zero or more gap bit sequences, andone or more gap size bit sequences that represent a portion of thesecond gap size, wherein the second gap size is equal to or greater thana threshold.
 6. The computer-readable media of claim 1, wherein thecompressed sets of one or more bit sequences include: a first map bitgroup that represents a first group of bit sequences from said groups ofbit sequences, wherein said first group has a first gap size, whereinsaid first map bit group contains zero or more map bit sequences andzero or more gap bit sequences, wherein said first gap size is less thana threshold; and a second map bit group that represents a second groupof bit sequences that has said second gap size, wherein said second mapbit group contains zero or more map bit sequences, zero or more gap bitsequences, and one or more gap size bit sequences that represent aportion of the second gap size, wherein the second gap size is equal toor greater than a threshold.
 7. The computer-readable media of claim 1,wherein at least one compressed set of said compressed sets containsanother set of one or more gap size bit sequences, wherein said otherset of one or more gap size bit sequences represents an offset number,wherein said offset number is less than the gap size of the group of bitsequences represented by said at least one compressed set that containssaid other set of one or more gap size bit sequences.
 8. Acomputer-readable media containing compressed output, comprising:compressed sets of one or more bit sequences that represent groups ofbit sequences; wherein each bit sequence of said groups of bit sequencescontains an identical number of bits; wherein each group of bitsequences of said groups of bit sequences includes either a gap bitsequence, an offset bit sequence, or a map bit sequence; and wherein atleast one compressed set of said compressed sets contains a set of oneor more gap size bit sequences, wherein said set of one or more gap sizebit sequences represents an offset number, wherein said offset number isless than the gap size of the group of bit sequences represented by saidat least one compressed set.
 9. The computer-readable media of claim 8,wherein at least one compressed set of one or more bit sequences of saidcompressed sets contains another set of one or more gap size bitsequences, wherein each gap size bit sequence of said other set of oneor more gap size bit sequences includes a flag, wherein said flagindicates whether a subsequent gap size bit sequence follows within saidat least one compressed set that contains said other set of one or moregap size bit sequences.
 10. The computer-readable media of claim 8,wherein wherein each gap bit sequence in said group of bit sequences hasall bits set to logical zero, wherein each offset bit sequence in saidgroup of bit sequences has one bit of said bit sequence set to logicalone, wherein each said map bit sequence in said group of bit sequenceshas two or more bits.
 11. The computer-readable media of claim 8,wherein each gap size sequence in said compressed sets of one or morebit sequences stores a portion of a gap size of the respective group ofbit sequences of said each gap size sequence.
 12. The computer-readablemedia of claim 8, wherein each compressed set of one or more bitsequences of said compressed sets includes a control bit sequence,wherein said control bit sequence indicates: a number of gap bitsequences in said each group, a number of map bit sequences in said eachgroup, and if said each group contains an offset bit sequence, thenwhich bit is set to value one in said offset bit sequence in said eachgroup.
 13. The computer-readable media of claim 8, wherein thecompressed sets of one or more bit sequences include: a first gap bitgroup that represents a first group of bit sequences from said groups ofbit sequences, wherein said first group has a first gap size, whereinsaid first gap bit group contains zero or more offset bit sequences andzero or more gap bit sequences, wherein said first gap size is less thana threshold; and a second gap bit group that represents a second groupof bit sequences that has a second gap size, wherein said second gap bitgroup contains zero or more offset bit sequences, zero or more gap bitsequences, and one or more gap size bit sequences that represent aportion of the second gap size, wherein the second gap size is equal toor greater than a threshold.
 14. The computer-readable media of claim 8,wherein the compressed sets of one or more bit sequences include: afirst map bit group that represents a first group of bit sequences fromsaid groups of bit sequences, wherein said first group has a first gapsize, wherein said first map bit group contains zero or more map bitsequences and zero or more gap bit sequences, wherein said first gapsize is less than a threshold; and a second map bit group thatrepresents a second group of bit sequences that has said second gapsize, wherein said second map bit group contains zero or more map bitsequences, zero or more gap bit sequences, and one or more gap size bitsequences that represent a portion of the second gap size, wherein thesecond gap size is equal to or greater than a threshold.
 15. Acomputer-readable media containing compressed output, comprising:compressed sets of one or more bit sequences that represent groups ofbit sequences; wherein each bit sequence of said groups of bit sequencescontains an identical number of bits; wherein each group of bitsequences of said groups of bit sequences includes either a gap bitsequence, an offset bit sequence, or a map bit sequence; and whereineach compressed set of one or more bit sequences of said compressed setscontains a set of one or more gap size bit sequences, wherein saidcontrol bit sequence contains a field representing a range of numbers,wherein said range of numbers includes a first subrange and a secondsubrange; wherein a first subgroup of said group of bit sequences has afirst bit sequence with one bit set to value one and said field set to aparticular value within said first subrange, wherein said valuerepresents the gap size of said group; and wherein a second subgroup ofsaid group of bit sequences has a second bit sequence with one bit setto value one and said field set to a particular value within said secondsubrange, wherein said value represents the gap size of said secondgroup, and wherein the gap size of said second subgroup and said firstsubgroup are the same.
 16. The computer-readable media of claim 15,wherein at least one compressed set of one or more bit sequences of saidcompressed sets contains a set of one or more gap size bit sequences,wherein each gap size bit sequence of said set of one or more gap sizebit sequences includes a flag, wherein said flag indicates whether asubsequent gap size bit sequence follows within said at least onecompressed set.
 17. The computer-readable media of claim 15, wherein atleast one compressed set of said compressed sets contains a set of oneor more gap size bit sequences, wherein said set of one or more gap sizebit sequences represents an offset number, wherein said offset number isless than the gap size of the group of bit sequences represented by saidat least one compressed set.
 18. The computer-readable media of claim15, wherein wherein each gap bit sequence in said group of bit sequenceshas all bits set to logical zero, wherein each offset bit sequence insaid group of bit sequences has one bit of said bit sequence set tological one, wherein each said map bit sequence in said group of bitsequences has two or more bits.
 19. The computer-readable media of claim15, wherein each gap size sequence in said compressed sets of one ormore bit sequences stores a portion of a gap size of the respectivegroup of bit sequences of said each gap size sequence.